Empirical Findings in Agile Methods

In recent years, the use of, interest in, and controversy about Agile methodologies have realized dramatic growth. Anecdotal evidence is rising regarding the effectiveness of agile methodologies in certain environments and for specified projects. However, collection and analysis of empirical evidence of this effectiveness and classification of appropriate environments for Agile projects has not been conducted. Researc hers from four institutions organized an eWorkshop to synchronously and virtually discuss and gather experiences and knowledge from eighteen Agile experts spread across the globe. These experts characterized Agile Methods and communicated experiences using these methods on small to very large teams. They discussed the importance of staffing Agile teams with highly skilled developers. They shared common success factors and identified warning signs of problems in Agile projects. These and other findings and heuristics gathered through this valuable exchange can be useful to researchers and to practitioners as they establish an experience base for better decision making. 1. The rise of Agile Methods Plan-driven methods are those in which work begins with the elicitation and documentation of a “complete” set of requirements, followed by architectural and high level-design development and inspection. Examples of plan-driven methods include various waterfall and iterative approaches, such as the Personal Software Process (PSP) [1]. Beginning in the mid-1990’s, some practitioners found these initial requirements documentation, and architecture and design development steps frustrating and, perhaps, impossible [2]. As Barry Boehm [3] suggests, these plan-driven methods may well start to pose difficulties when change rates are still relatively low. The industry and the technology move too fast and customers have become increasingly unable to definitively state their needs up front. As a result, several consultants have independently developed methodologies and practices to embrace and respond to the inevitable change they were experiencing. These methodologies and practices are based on iterative enhancement, a technique which was introduced in 1975 [4] and that has been come to be known as Agile Methodologies [2, 5]. Agile Methodologies are gaining popularity in industry although they comprise a mix of accepted and controversial software engineering practices. It is quite likely that the software industry will find that specific project characteristics will determine the prudence of using an agile or a plan-driven methodology – or a hybrid of the two. In recent years, there have been many stories and anecdotes [6-8] of industrial teams experiencing success with Agile methodologies. There is, however, an urgent need to empirically assess the applicability of these methods, in a structured manner, in order to build an experience base for better decision-making. This paper contributes to the experience base and discusses the findings of a synchronous, virtual eWorkshop in which experiences and knowledge were gathered from and shared between Agile experts located throughout the world. 2. An experience base for software engineering In order to reach their goals, software development teams need to understand and choose the right models and techniques to support their projects. They must consider key questions such as: What is the best life-cycle model to choose for a particular project? What is an appropriate balance of effort between documenting the work and ge tting the product implemented? When does it pay-off to spend major efforts on planning in advance and avoid change, and when is it more beneficial to plan less rigorously and embrace change? The goal of the NSF-sponsored Center for Empirically-Based Software Engineering (CeBASE)1 is to collect, analyze, document, and disseminate knowledge on software engineering gained from experiments, case studies, observations, interviews, expert discussions and real world projects. A central activity toward achieving this goal has been the running of “eWorkshops” (or online meetings) that capture expert knowledge to formulate heuristics on a particular software engineering topic. The CeBASE project defined the eWorkshop and has used the technology to collect valuable empirical evidence on defect reduction and COTS. [9] The rise of Agile Methods provides a fruitful area for such empirical research. This paper discusses the results of the first eWorkshop on Agile Methods sponsored by the Fraunhofer Center Maryland and North Carolina State University using the CeBASE eWorkshop technology. The discussion items are presented along with an encapsulated summary of the expert discussion. The heuristics can be useful both to researchers (for pointing out gaps in the current state of the knowledge requiring further investigation) and to practitioners (for benchmarking or setting expectations about development practices). 1 http://www.CeBASE.org 3. Collecting expert knowledge on Agile methods Workshops in which experts discuss their findings and record their discussions are a classic method for creating and disseminating knowledge. Workshops, however, possess limitations: 1) experts are spread all over the world and would have to travel to participate, and 2) workshops are usually oral presentations and discussions, which are generally not captured for further analysis. The eWorkshops are designed to help overcome these problems. The eWorkshop is an on-line meeting that replaces the usual face-to-face workshop. While it uses a Web-based chat-application, the session is structured to accommodate the needs of a workshop without becoming an unconstrained online chat discussion. The goal of the eWorkshop is to synthesize new knowledge from a group of experts in an efficient and inexpensive manner in order to populate an experience base. More details about the eWorkshop tool and process can be found in [10]. The goal of the Agile workshop, held in April 2002, was to create a set of heuristics that represent what experts in the field consider to be the current state of understanding about Agile Methods. The participants in this event were experts in Agile Methods. Our lead discussants (the workshop leaders and authors of this paper) formed part of the team that interacted with an international group of invited participant experts. The names of these 18 participants are listed in the acknowledgements at the end of the paper. 4. Seeding the eDiscussion Participants of eWorkshops prepare for the discussion by reading relevant material, preparing a position statement reacting to proposed discussion points, and reading the position statements of the other discussants. For this eWorkshop, Barry Boehm’s January 2002 IEEE Computer article [3] and the Agile Manifesto [11-13] served as background material. The Agile Manifesto documents the priorities of its signers. They value: • Individuals and interactions over processes and tools • Working software over comprehensive documentation • Customer collaboration over contract negotiation • Responding to change over following a plan Many in industry debate the prudence of these values. Steven Rakitin comments [14] that the items on the right are essential, while those on the left only serve as excuses for hackers to keep on irresponsibly throwing code together with no regard for engineering discipline. Another example is Matt Stephens’ critical analysis of XP and its applicability2. It is important to remember that Agile includes many different methodologies, of which the best known include: • Extreme Programming (XP) [15-17]